A New Very-General Method to Generate Random Modal Formulae for Testing Decision Procedures

The recent emergence of heavily-optimized modal decision procedures has highlighted the key role of empirical testing in this domain. Unfortunately, the introduction of extensive empirical tests for modal logics is recent, and so far none of the proposed test generators is very satisfactory. To cope with this fact, we present a new random generation method that provides benefits over previous methods for generating empirical tests. It fixes and much generalizes one of the best-known methods, the random CNF2m test, allowing for generating a much wider variety of problems, covering in principle the whole input space. Our new method produces much more suitable test sets for the current generation of modal decision procedures. We analyze the features of the new method by means of an extensive collection of empirical tests. 1. Motivation and goals Heavily-optimized systems for determining satisfiability of formulae in propositional modal logics are becoming available. These systems, including DLP (Patel-Schneider, 1998), FaCT (Horrocks, 1998), *SAT (Giunchiglia, Giunchiglia, & Tacchella, 2001), MSPASS (Hustadt, Schmidt, & Weidenbach, 1999), and RACER (Haarslev & Moller, 2001), have more optimizations and are much faster than the previous generation of modal decision procedures, such as leanK (Beckert & Goré, 1997), Logics Workbench (Heuerding, Jäger, Schwendimann, & Seyfreid, 1995), 2KE (Pitt & Cunningham, 1996) and KSAT (Giunchiglia & Sebastiani, 2000). 1 As with most theorem proving problems, neither computational complexity nor asymptotic algorithmic complexity is very useful in determining the effectiveness of optimizations, so that their effectiveness has to be determined by empirical testing (Horrocks, Patel-Schneider, & Sebastiani, 2000). Empirical testing directly gives resource consumption in terms of compute time and memory use; it factors in all the pieces of the system, not just the basic algorithm itself. Empirical testing can be used not only to compare different systems, but also to tune a system with parameters that can be used to modify its performance; moreover, it can be used to show what sort of inputs the system handles well, and what sort of inputs the system handles poorly. Unfortunately, the introduction of extensive empirical tests for modal logics is recent, and so far none of the proposed test methodologies are very satisfactory. Some methods contain many formulae that are too easy for current heavily-optimized procedures. Some contain high rates of trivial or insignificant tests. Some generate problems that are too artificial and/or are not a significant 1. For a more complete list see, e.g., Renate Schmidt’s page on theorem provers for modal logics at http://www.cs.man.ac.uk/ ̃schmidt/tools/. c AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved. PATEL-SCHNEIDER AND SEBASTIANI sample of the input space. Finally, some methods generate formulae that are too big to be parsed and/or handled. For the reasons described above, we presented (Horrocks et al., 2000) an analytical survey of the state-of-the art of empirical testing for modal decision procedures. Here instead we present a new random generation method that provides benefits over previous methods for generating empirical tests, built on some preliminary work (Horrocks et al., 2000). Our new method fixes and much generalizes the 3CNF2m methodology for randomly generating clausal formulae in modal logics (Giunchiglia & Sebastiani, 1996; Hustadt & Schmidt, 1999; Giunchiglia, Giunchiglia, Sebastiani, & Tacchella, 2000) used in many previous empirical tests of modal decision procedures. It eliminates or drastically reduces the influence of a major flaw of the previous method, and allows for generating a much wider variety of problems. In Section 2 we recall a list of desirable features for good test sets. In Section 3 we briefly survey the state-of-the-art test methods. In Sections 4 and 5 we present and discuss the basic and the advanced versions of our new test method respectively, and evaluate their features by presenting a big amount of empirical results. In Section 6 we provide a theoretical result showing how the advanced version of our method, in principle, can cover the whole input space. In Section 7 we discuss the features of our new method, and compare it wrt. the state-of-the-art methods. In Section 8 we conclude and indicate possible future research directions. A 5-page system description of our random generator has been presented at IJCAR’2001 (PatelSchneider & Sebastiani, 2001). 2. Desirable features for good test sets The benefits of empirical testing depend on the characteristics of the inputs provided for the testing, as empirical testing only provides data on these particular inputs. If the inputs are not typical or suitable, then the results of the empirical testing will not be useful. This means that the inputs for empirical testing must be carefully chosen. We have previously proposed and motivated the following key criteria for creating good test sets (Horrocks et al., 2000). Significance. The ideal test set should represent a significant sample of the whole input space. A good empirical test set should at least cover a large area of inputs. Difficulty. A good empirical test set should provide a sufficient level of difficulty for the system(s) being tested. (Some problems should be too hard even for state-of-the-art systems, so as to be a good benchmark for forthcoming systems.) Termination. To be of practical use, the tests should terminate and provide information within a reasonable amount of time. Scalability. The difficulty of problems should scale up, as comparing absolute performances may be less significant than comparing how performances scale up with problems of increasing difficulty. Valid vs. not-valid balance. In a good test set, valid and not-valid problems should be more or less equal both in number and in difficulty. Moreover, the maximum uncertainty regarding the solution of the problems is desirable.